Add Azurite test support for AzureBlobStore by Muktarsadiq · Pull Request #2684 · TraceMachina/nativelink

Muktarsadiq · 2026-08-12T19:43:23Z

Summary

Adds Azurite (Azure Storage emulator) support to the test suite for
AzureBlobStore, Follows the same embedded-runner pattern
as mongo_runner, adapted for Azurite's constraints.

Description

Adds an embedded Azurite (Azure Storage emulator) test runner, mirroring the existing mongo_runner pattern.

Azurite has no standalone binary distribution, so the runner invokes a locally npm-installed azurite-blob directly rather than downloading one. The store is pointed at it via ExperimentalAzureSpec.sas_url rather than endpoint, since endpoint alone routes through WorkloadIdentityCredential, which Azurite cannot satisfy. A SAS signer generates both a container scoped Service SAS for test operations and an Account SAS for one time container bootstrapping, since a container SAS cannot authorize creating the container it names.

Also fixes a process crash discovered while building this: Azurite logs every request to stdout, and reading that pipe only long enough to capture the startup port left it undrained afterward, causing the next write to block and crash the whole process. stdout is now drained for the process's full lifetime instead.

Fixes #2511

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Added upload_and_get_data, upload_empty_data, and
zero_len_items_exist_check, run against a real AzureBlobStore
talking to a locally-spawned Azurite instance — no mocking. Two
#[ignore]d tests document the manual verification steps used to
validate the SAS signer and container-bootstrap logic independently
before wiring them into the automatic flow.

Note on formatting/testing: this environment doesn't have Nix
available (macOS 12/Monterey isn't supported by the current Nix
installer), so bazel test //... and the Bazel-driven nightly
rustfmt target couldn't be run directly. Ran cargo +nightly fmt
locally as the closest available equivalent, and verified everything
via cargo build --all --profile=smol / cargo test --all --profile=smol, matching what native-cargo.yaml's CI job runs.

Local setup:

cd nativelink-store/tests/azurite_runner && npm ci
cargo test -p nativelink-store --test azurite_store_test

CI: added actions/setup-node (pinned to v7.0.0) + npm ci to
native-cargo.yaml, on both OS legs, cached on
nativelink-store/tests/azurite_runner/package-lock.json.

Checklist

Updated documentation if needed
Tests added/amended
bazel test //... passes locally
PR is contained in a single commit, using git amend see some docs

This change is

vercel · 2026-08-12T19:43:29Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview	Aug 24, 2026 4:35pm
nativelink-aidm	Ready	Preview	Aug 24, 2026 4:35pm

CLAassistant · 2026-08-12T19:43:29Z

All committers have signed the CLA.

MarcusSorealheis · 2026-08-12T22:10:09Z

Thank you for the PR. We will check it out soon!

palfrey

Thanks for doing this. There's a few issues though.

palfrey · 2026-08-19T16:03:00Z

-      - '.github/styles/**'
-      - 'web/**'
+      - ".github/styles/**"
+      - "web/**"


Can you please reset the format changes here

palfrey · 2026-08-19T16:04:34Z

+
+      - name: Install Azurite for Azure store tests
+        run: npm ci
+        working-directory: nativelink-store/tests/azurite_runner


Please use bun for package management, as per our existing other usage in https://github.com/TraceMachina/nativelink/blob/main/web/bun.lock

Also, please make these steps a new reusable action, as they'll need to get added to various other workflows e.g. .github/workflows/nix.yaml and .github/workflows/coverage.yaml

For nix.yaml's nix-cargo job, i added the composite action that works cleanly since it runs inside nix develop (a shell, not a sandbox). on the other hand coverage.yaml's job runs via nix build, a fully sandboxed derivation with no network access, the same reason mongo_runner needs the p.mongodb symlink hack in flake.nix. Azurite isn't a nixpkgs-native package the way MongoDB is, so the same trick isn't directly available. Happy to look into packaging Azurite for Nix if that's the preferred path, but wanted to flag this as a separate problem rather than silently skip it or fake a fix.

palfrey · 2026-08-19T16:05:25Z

+tokio = { version = "1.52.2", features = [
+  "process",
+  "sync",
+], default-features = false }


Please move these further down. Our standard formatting for dependencies is the nativelink internal ones at the top, then everything else in alphabetical order after that.

palfrey · 2026-08-19T16:11:31Z

+  "dependencies": {
+    "azurite": "3.36.0"
+  }
+}


This needs reformatting with the pre-commit rules. If you run the Nix setup you'll have pre-commit setup correctly as part of that.

palfrey · 2026-08-19T16:13:32Z

This needs adding to the test suite in nativelink-store/BUILD.bazel

Added tests/azurite_store_test.rs to the suite, plus the needed compile_data/deps. Same sandboxing caveat as my note on the coverage.yaml comment. I suspect bazel test's sandbox has the same no network issue, so I can't verify this actually passes at runtime, only that it should compile. Happy to dig further if useful.

github-actions · 2026-08-24T11:12:31Z

Some of the pull request description still needs filling in:

What and why is missing. Please keep the template's headings.
How was this verified? is missing. Please keep the template's headings.
Risk is missing. Please keep the template's headings.

Edit the description and this check re-runs on its own. The sections exist because they are the parts a reviewer cannot get from the diff: why the change is needed, how you know it works, and what breaks if it is wrong.

Adds an embedded Azurite (Azure Storage emulator) test runner, mirroring the existing mongo_runner pattern, and closes TraceMachina#2511. Azurite has no standalone binary distribution, so the runner invokes a locally npm-installed azurite-blob directly rather than downloading one. The store is pointed at it via ExperimentalAzureSpec.sas_url rather than endpoint, since endpoint alone routes through WorkloadIdentityCredential, which Azurite cannot satisfy. A SAS signer generates both a container scoped Service SAS for test operations and an Account SAS for one time container bootstrapping, since a container SAS cannot authorize creating the container it names. Also fixes a process crash discovered while building this: Azurite logs every request to stdout, and reading that pipe only long enough to capture the startup port left it undrained afterward, causing the next write to block and crash the whole process. stdout is now drained for the process's full lifetime instead. CI: adds actions/setup-node and npm ci to native-cargo.yaml so azurite-blob is available on both OS legs before cargo test runs.

palfrey · 2026-08-25T11:39:03Z

I would suggest running all of this locally under Bazel with nix, so you can fix the variety of lint failures there as well as run pre-commit run -a to fix all the pre-commit issues currently being flagged in the build.

vercel Bot deployed to Preview – nativelink-aidm August 12, 2026 19:43 View deployment

vercel Bot deployed to Preview – nativelink August 12, 2026 19:44 View deployment

vercel Bot deployed to Preview – nativelink-aidm August 12, 2026 20:10 View deployment

vercel Bot deployed to Preview – nativelink August 12, 2026 20:11 View deployment

vercel Bot deployed to Preview – nativelink-aidm August 13, 2026 09:41 View deployment

vercel Bot deployed to Preview – nativelink August 13, 2026 09:41 View deployment

Muktarsadiq force-pushed the azurite-test-support branch from d35c7e3 to 419234e Compare August 13, 2026 19:43

vercel Bot deployed to Preview – nativelink-aidm August 13, 2026 19:44 View deployment

vercel Bot deployed to Preview – nativelink August 13, 2026 19:44 View deployment

palfrey requested changes Aug 19, 2026

View reviewed changes

Muktarsadiq force-pushed the azurite-test-support branch from 419234e to 6a12eca Compare August 24, 2026 11:12

vercel Bot deployed to Preview – nativelink-aidm August 24, 2026 11:12 View deployment

vercel Bot deployed to Preview – nativelink August 24, 2026 11:13 View deployment

palfrey reviewed Aug 24, 2026

View reviewed changes

Comment thread nativelink-store/BUILD.bazel Outdated

Muktarsadiq force-pushed the azurite-test-support branch from 6a12eca to 5ee429d Compare August 24, 2026 16:35

vercel Bot deployed to Preview – nativelink-aidm August 24, 2026 16:35 View deployment

vercel Bot deployed to Preview – nativelink August 24, 2026 16:35 View deployment

Uh oh!

Conversation

Muktarsadiq commented Aug 12, 2026 • edited by MarcusSorealheis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Description

Type of change

How Has This Been Tested?

Checklist

Uh oh!

vercel Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcusSorealheis commented Aug 12, 2026

Uh oh!

palfrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 24, 2026

Uh oh!

Uh oh!

palfrey commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Muktarsadiq commented Aug 12, 2026 •

edited by MarcusSorealheis

Loading

vercel Bot commented Aug 12, 2026 •

edited

Loading

CLAassistant commented Aug 12, 2026 •

edited

Loading